Inbar Naus 312231665 Barak Samesh 312361041

Assignment 1 – Practical Deep Learning Workshop


Part 1- Exploring the Dataset

We choose the CIFAR-100 Dataset:

a. The size of the Dataset is 60,000 photos devided to 50,000 train samples and 10,000 test samples.

b. Each sample of data is 32x32 pixels, and 3 channels (RGB). There are 20 superclasses and for each superclass there are 5 classes.

c. There is no need for preproccessing our dataset. We can use horizontal flipping and gentle rotation, for example, for enlarging our dataset.

d. The data is balanced. (look at the graph in cell 3).

e. Using EfficientNet 91.7% was achived on CIFAR-100.

In [1]:
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
from matplotlib import pyplot as plt # chart plotting
from sklearn.metrics import accuracy_score, confusion_matrix

import seaborn as sns # plotting heat maps

from keras.datasets import cifar100
from keras.models import Model, Sequential
from keras.layers import Dense, Conv2D, Flatten, Input, AvgPool2D, MaxPool2D, Dropout, BatchNormalization, GlobalAveragePooling2D
from keras.utils import to_categorical
from keras.callbacks import EarlyStopping, ReduceLROnPlateau

import warnings
from functools import reduce  # functional operation on lists
import math
import os
/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm
Using TensorFlow backend.
In [2]:
# loading the data
(X_train, y_train), (X_test, y_test) = cifar100.load_data()
Downloading data from https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz
169009152/169001437 [==============================] - 13s 0us/step
In [27]:
sorted = np.sort(np.concatenate((y_test, y_train), axis=None), axis = 0)
unique_elements, counts_elements = np.unique(y_train, return_counts=True)

plt.figure(figsize=(20,5))
# plotting a bar chart 
plt.bar(range(len(unique_elements)), counts_elements, 
        width = 0.3, color = ['blue']) 
  
# naming the x-axis 
plt.xlabel('label indices') 
# naming the y-axis 
plt.ylabel('amount of images for each label') 
# plot title 
plt.title('Histogram of labels in the dataset') 

# function to show the plot 
plt.show()
In [6]:
label_list = [
    'apple', 'aquarium_fish', 'baby', 'bear', 'beaver', 'bed', 'bee', 'beetle', 
    'bicycle', 'bottle', 'bowl', 'boy', 'bridge', 'bus', 'butterfly', 'camel', 
    'can', 'castle', 'caterpillar', 'cattle', 'chair', 'chimpanzee', 'clock', 
    'cloud', 'cockroach', 'couch', 'crab', 'crocodile', 'cup', 'dinosaur', 
    'dolphin', 'elephant', 'flatfish', 'forest', 'fox', 'girl', 'hamster', 
    'house', 'kangaroo', 'keyboard', 'lamp', 'lawn_mower', 'leopard', 'lion',
    'lizard', 'lobster', 'man', 'maple', 'motorcycle', 'mountain', 'mouse',
    'mushroom', 'oak', 'orange', 'orchid', 'otter', 'palm', 'pear',
    'pickup_truck', 'pine', 'plain', 'plate', 'poppie', 'porcupine',
    'possum', 'rabbit', 'raccoon', 'ray', 'road', 'rocket', 'rose',
    'sea', 'seal', 'shark', 'shrew', 'skunk', 'skyscraper', 'snail', 'snake',
    'spider', 'squirrel', 'streetcar', 'sunflower', 'sweet_pepper', 'table',
    'tank', 'telephone', 'television', 'tiger', 'tractor', 'train', 'trout',
    'tulip', 'turtle', 'wardrobe', 'whale', 'willow', 'wolf', 'woman',
    'worm']

unique = []
images = []

for x in range(X_train.size):
  if len(unique) < 100 :
    if y_train[x].item(0) not in unique:
        unique.append(y_train[x].item(0))
        images.append(X_train[x])
  else:
      break

labels = []
for x in unique:
  labels.append(label_list[x])

_, axes = plt.subplots(10, 10, figsize=(20, 20))
index = 0
for ax, img ,title in zip(axes.flatten(), images, labels):
    ax.imshow(img)
    ax.set_title(title)
    ax.set_xticks([])
    ax.set_yticks([])
    index = index + 5

Part 2- Building and Fitting a Model

a. We have decided to use cross-validation strategy for training our model.

b. The model's construction, train and results analysis:

In [0]:
model = Sequential()
model.add(Conv2D(64,(3,3),activation='relu',input_shape=(32,32,3)))
model.add(Conv2D(128,(1,1), activation='relu' ))
model.add(Dropout(0.3))

model.add(AvgPool2D(2,2))
model.add(Conv2D(128,(3,3), activation='relu' ))
model.add(Conv2D(192,(3,3), activation='relu' ))
model.add(Dropout(0.4))

model.add(AvgPool2D(2,2))
model.add(Conv2D(192,(3,3), activation='relu' ))
model.add(Conv2D(192,(3,3), activation='relu' ))
model.add(Dropout(0.5))

model.add(Flatten())
model.add(Dense(100,activation='softmax'))

model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])

print(model.summary())
Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_13 (Conv2D)           (None, 30, 30, 64)        1792      
_________________________________________________________________
conv2d_14 (Conv2D)           (None, 30, 30, 128)       8320      
_________________________________________________________________
dropout_7 (Dropout)          (None, 30, 30, 128)       0         
_________________________________________________________________
average_pooling2d_5 (Average (None, 15, 15, 128)       0         
_________________________________________________________________
conv2d_15 (Conv2D)           (None, 13, 13, 128)       147584    
_________________________________________________________________
conv2d_16 (Conv2D)           (None, 11, 11, 192)       221376    
_________________________________________________________________
dropout_8 (Dropout)          (None, 11, 11, 192)       0         
_________________________________________________________________
average_pooling2d_6 (Average (None, 5, 5, 192)         0         
_________________________________________________________________
conv2d_17 (Conv2D)           (None, 3, 3, 192)         331968    
_________________________________________________________________
conv2d_18 (Conv2D)           (None, 1, 1, 192)         331968    
_________________________________________________________________
dropout_9 (Dropout)          (None, 1, 1, 192)         0         
_________________________________________________________________
flatten_3 (Flatten)          (None, 192)               0         
_________________________________________________________________
dense_3 (Dense)              (None, 100)               19300     
=================================================================
Total params: 1,062,308
Trainable params: 1,062,308
Non-trainable params: 0
_________________________________________________________________
None
In [0]:
# training the model

monitor_val_acc = EarlyStopping(monitor='val_accuracy', patience=6)
lr_sched = ReduceLROnPlateau(patience=4)


history = model.fit(X_train,
    to_categorical(y_train),
    batch_size=120,
    epochs=50,
    verbose=1,
    shuffle=True,
    callbacks=[monitor_val_acc, lr_sched],
    validation_split=0.2)
Train on 40000 samples, validate on 10000 samples
Epoch 1/50
40000/40000 [==============================] - 7s 170us/step - loss: 4.4662 - accuracy: 0.0303 - val_loss: 4.1281 - val_accuracy: 0.0709
Epoch 2/50
40000/40000 [==============================] - 6s 162us/step - loss: 4.0563 - accuracy: 0.0789 - val_loss: 3.7639 - val_accuracy: 0.1330
Epoch 3/50
40000/40000 [==============================] - 7s 163us/step - loss: 3.7734 - accuracy: 0.1213 - val_loss: 3.5188 - val_accuracy: 0.1750
Epoch 4/50
40000/40000 [==============================] - 7s 163us/step - loss: 3.5432 - accuracy: 0.1596 - val_loss: 3.2789 - val_accuracy: 0.2145
Epoch 5/50
40000/40000 [==============================] - 7s 164us/step - loss: 3.3663 - accuracy: 0.1927 - val_loss: 3.1120 - val_accuracy: 0.2440
Epoch 6/50
40000/40000 [==============================] - 7s 164us/step - loss: 3.1900 - accuracy: 0.2232 - val_loss: 3.0180 - val_accuracy: 0.2574
Epoch 7/50
40000/40000 [==============================] - 7s 163us/step - loss: 3.0648 - accuracy: 0.2450 - val_loss: 2.9234 - val_accuracy: 0.2869
Epoch 8/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.9311 - accuracy: 0.2739 - val_loss: 2.8122 - val_accuracy: 0.3030
Epoch 9/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.8345 - accuracy: 0.2910 - val_loss: 2.7756 - val_accuracy: 0.3113
Epoch 10/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.7200 - accuracy: 0.3103 - val_loss: 2.6810 - val_accuracy: 0.3303
Epoch 11/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.6221 - accuracy: 0.3295 - val_loss: 2.7287 - val_accuracy: 0.3259
Epoch 12/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.5370 - accuracy: 0.3459 - val_loss: 2.5962 - val_accuracy: 0.3484
Epoch 13/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.4641 - accuracy: 0.3623 - val_loss: 2.6630 - val_accuracy: 0.3437
Epoch 14/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.3778 - accuracy: 0.3749 - val_loss: 2.5938 - val_accuracy: 0.3606
Epoch 15/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.3249 - accuracy: 0.3880 - val_loss: 2.5445 - val_accuracy: 0.3697
Epoch 16/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.2410 - accuracy: 0.4079 - val_loss: 2.5459 - val_accuracy: 0.3648
Epoch 17/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.1997 - accuracy: 0.4162 - val_loss: 2.5239 - val_accuracy: 0.3738
Epoch 18/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.1337 - accuracy: 0.4270 - val_loss: 2.5736 - val_accuracy: 0.3706
Epoch 19/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.0844 - accuracy: 0.4399 - val_loss: 2.5651 - val_accuracy: 0.3735
Epoch 20/50
40000/40000 [==============================] - 7s 163us/step - loss: 2.0118 - accuracy: 0.4532 - val_loss: 2.5431 - val_accuracy: 0.3821
Epoch 21/50
40000/40000 [==============================] - 7s 163us/step - loss: 1.9696 - accuracy: 0.4642 - val_loss: 2.5936 - val_accuracy: 0.3798
Epoch 22/50
40000/40000 [==============================] - 7s 163us/step - loss: 1.6859 - accuracy: 0.5228 - val_loss: 2.5438 - val_accuracy: 0.4012
Epoch 23/50
40000/40000 [==============================] - 7s 163us/step - loss: 1.6047 - accuracy: 0.5433 - val_loss: 2.5394 - val_accuracy: 0.4030
Epoch 24/50
40000/40000 [==============================] - 7s 163us/step - loss: 1.5523 - accuracy: 0.5550 - val_loss: 2.5654 - val_accuracy: 0.4017
Epoch 25/50
40000/40000 [==============================] - 7s 163us/step - loss: 1.5208 - accuracy: 0.5613 - val_loss: 2.5745 - val_accuracy: 0.4030
Epoch 26/50
40000/40000 [==============================] - 6s 162us/step - loss: 1.4868 - accuracy: 0.5690 - val_loss: 2.5700 - val_accuracy: 0.4055
Epoch 27/50
40000/40000 [==============================] - 6s 162us/step - loss: 1.4821 - accuracy: 0.5703 - val_loss: 2.5741 - val_accuracy: 0.4056
Epoch 28/50
40000/40000 [==============================] - 7s 163us/step - loss: 1.4800 - accuracy: 0.5704 - val_loss: 2.5760 - val_accuracy: 0.4057
Epoch 29/50
40000/40000 [==============================] - 7s 163us/step - loss: 1.4730 - accuracy: 0.5715 - val_loss: 2.5769 - val_accuracy: 0.4043
Epoch 30/50
40000/40000 [==============================] - 7s 164us/step - loss: 1.4619 - accuracy: 0.5770 - val_loss: 2.5765 - val_accuracy: 0.4046
Epoch 31/50
40000/40000 [==============================] - 7s 163us/step - loss: 1.4671 - accuracy: 0.5722 - val_loss: 2.5769 - val_accuracy: 0.4046
Epoch 32/50
40000/40000 [==============================] - 6s 162us/step - loss: 1.4643 - accuracy: 0.5735 - val_loss: 2.5768 - val_accuracy: 0.4052
Epoch 33/50
40000/40000 [==============================] - 6s 162us/step - loss: 1.4670 - accuracy: 0.5756 - val_loss: 2.5773 - val_accuracy: 0.4049
Epoch 34/50
40000/40000 [==============================] - 7s 164us/step - loss: 1.4663 - accuracy: 0.5734 - val_loss: 2.5773 - val_accuracy: 0.4048
In [0]:
# Plot training & validation accuracy rates
fig, ax = plt.subplots(1,2,figsize=(12,4))
ax[0].plot(history.history['accuracy'])
ax[0].plot(history.history['val_accuracy'])
ax[0].set_title('Model Accuracy')
ax[0].set_ylabel('Accuracy')
ax[0].set_xlabel('Epoch')
ax[0].legend(['Train', 'Test'], loc='upper left')

# Plot training & validation loss rates
ax[1].plot(history.history['loss'])
ax[1].plot(history.history['val_loss'])
ax[1].set_title('Model Loss')
ax[1].set_ylabel('Loss')
ax[1].set_xlabel('Epoch')
ax[1].legend(['Train', 'Test'], loc='upper left')
plt.show()

The model achieved poor accuracy percentage compared to the benchmark mentioned above.

It is also noticable that the model suffers from overfitting.

In [0]:
# plot a heat map of the predictions on unseen data

y_pred = model.predict(X_test)
pred_cat = np.argmax(y_pred,axis=1)
confusion_mat = confusion_matrix(y_test,pred_cat)
print('model accuracy on test set is: {}%'.format(accuracy_score(y_test,pred_cat)*100))
sns.heatmap(confusion_mat,cmap='Greens',annot=True, fmt='d')
sns.set(rc={'figure.figsize':(50,30)})
plt.xlabel('Prediction')
plt.ylabel('True label')
plt.title('First model results on test set')
model accuracy on test set is: 40.92%
Out[0]:
Text(0.5, 1.0, 'First model results on test set')

We can see that the label that was predicted by the model correctly the most times is Plain.

Below are a few examples of samples labled 'Plain'.

In [18]:
#picking the class with most right predictions (highest value on the main diagonal)
diagonal = [confusion_mat[i][i] for i in range(len(confusion_mat)) ]
maximum_value = reduce((lambda element, maximum: max(maximum, element)), diagonal)
class_index = diagonal.index(maximum_value) # the index of the desired class

def plot_samples(class_index, n_samples):
  """A function that prints n_samples image samples from the class with index class_index"""
  # finding the indices of samples labeled label_index
  sample_indices = np.where(y_test == class_index)
  samples = []
  # sampling
  for x in range(n_samples):
    samples.append(X_test[sample_indices[0][x]])

  #creating titles list
  title = label_list[class_index]
  titles = [title + " " + str(x+1) for x in range(n_samples)]
  # plotting
  _, axes = plt.subplots(1, n_samples, figsize=(10,10))
  for ax, img, title in zip(axes.flatten(), samples, titles):
      ax.imshow(img)
      ax.set_title(title)
      ax.set_xticks([])
      ax.set_yticks([])

plot_samples(class_index, 4)

C. We can see that a common mistake the model made is predicting Willow to be Oak. If we examin samples from the corresponding classes, we can indeed notice some visual similarities (well, they both are trees..).

Below are a few examples for samples from the corresponding classes.

In [19]:
# picking a common mistake and plotting corresponding samples

pred_index = 52 # oak
true_index = 96 # willow

#print samples of the relevant classes
plot_samples(pred_index, 3)
plot_samples(true_index, 3)

d. We can improve the accuracy by taking these three meassures:

  • Changing architecture
  • Augmentation
  • K-Fold cross validation

We will implement the first two of these. First, changing the architecture, including usage of a new layer- BatchNormalization:

In [0]:
#model improvement
from keras import regularizers
weight_decay = 1e-5
model2 = Sequential()
model2.add(Conv2D(64,(3,3),activation='elu',input_shape=(32,32,3)))

model2.add(MaxPool2D(pool_size=(2,2)))
model2.add(Conv2D(92,(3,3),activation='elu', padding='same'))
model2.add(BatchNormalization())
model2.add(Dropout(0.4))

model2.add(Conv2D(128,(3,3),activation='elu', padding='same'))
model2.add(BatchNormalization())
model2.add(Dropout(0.4))

model2.add(MaxPool2D(pool_size=(2,2)))
model2.add(Conv2D(128,(3,3),activation='elu', padding='same'))
model2.add(BatchNormalization())
model2.add(Dropout(0.4))

model2.add(MaxPool2D(pool_size=(2,2)))
model2.add(Conv2D(128,(3,3),activation='elu', padding='same', kernel_regularizer=regularizers.l2(weight_decay)))
model2.add(BatchNormalization())
model2.add(Dropout(0.5))

model2.add(Flatten())
model2.add(Dense(912,activation='relu'))
model2.add(Dropout(0.4))
model2.add(Dense(100,activation='softmax'))

model2.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])

Next, using augmentation as preprocessing for enlarging the dataset:

In [0]:
from keras.preprocessing.image import ImageDataGenerator
import math

# Instantiating a data generator
datagen = ImageDataGenerator(
    rotation_range=10,
    width_shift_range=0.1,
    height_shift_range=0.1,
    horizontal_flip=True)

# determine the operation's parameters
batch_size = 10
train_val_split = 0.2
split_index = math.ceil(len(X_train) * (1 - train_val_split))
augmnt_factor = 2 # the factor of wich the dataset will be increased by 

train_iter = datagen.flow(X_train[:split_index],to_categorical(y_train[:split_index]), batch_size=batch_size)

val_iter = datagen.flow(X_train[split_index:],to_categorical(y_train[split_index:]), batch_size=batch_size)

history2 = model2.fit_generator(train_iter,
                    validation_data = val_iter,
                    steps_per_epoch = math.ceil(split_index / batch_size) * augmnt_factor,
                    validation_steps = math.ceil((len(X_train)-split_index) / batch_size) * augmnt_factor,
                    callbacks=[monitor_val_acc, lr_sched],
                    epochs=50)
Epoch 1/50
8000/8000 [==============================] - 111s 14ms/step - loss: 4.2306 - accuracy: 0.0573 - val_loss: 98.8827 - val_accuracy: 0.0487
Epoch 2/50
8000/8000 [==============================] - 110s 14ms/step - loss: 3.8500 - accuracy: 0.1043 - val_loss: 299.2845 - val_accuracy: 0.0848
Epoch 3/50
8000/8000 [==============================] - 108s 13ms/step - loss: 3.5473 - accuracy: 0.1555 - val_loss: 104.5778 - val_accuracy: 0.1338
Epoch 4/50
8000/8000 [==============================] - 107s 13ms/step - loss: 3.2872 - accuracy: 0.2030 - val_loss: 267.0200 - val_accuracy: 0.1297
Epoch 5/50
8000/8000 [==============================] - 109s 14ms/step - loss: 3.0849 - accuracy: 0.2430 - val_loss: 49.2853 - val_accuracy: 0.1705
Epoch 6/50
8000/8000 [==============================] - 109s 14ms/step - loss: 2.9556 - accuracy: 0.2715 - val_loss: 53.9747 - val_accuracy: 0.1546
Epoch 7/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.8637 - accuracy: 0.2898 - val_loss: 7.5935 - val_accuracy: 0.2490
Epoch 8/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.7928 - accuracy: 0.3064 - val_loss: 9.2342 - val_accuracy: 0.2444
Epoch 9/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.7420 - accuracy: 0.3177 - val_loss: 11.1257 - val_accuracy: 0.2769
Epoch 10/50
8000/8000 [==============================] - 109s 14ms/step - loss: 2.6920 - accuracy: 0.3272 - val_loss: 3.4110 - val_accuracy: 0.2725
Epoch 11/50
8000/8000 [==============================] - 107s 13ms/step - loss: 2.6488 - accuracy: 0.3372 - val_loss: 6.5235 - val_accuracy: 0.2304
Epoch 12/50
8000/8000 [==============================] - 107s 13ms/step - loss: 2.6272 - accuracy: 0.3423 - val_loss: 7.0151 - val_accuracy: 0.2478
Epoch 13/50
8000/8000 [==============================] - 108s 14ms/step - loss: 2.5932 - accuracy: 0.3502 - val_loss: 10.8707 - val_accuracy: 0.2854
Epoch 14/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.5698 - accuracy: 0.3568 - val_loss: 3.2281 - val_accuracy: 0.3153
Epoch 15/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.5556 - accuracy: 0.3600 - val_loss: 2.7215 - val_accuracy: 0.3292
Epoch 16/50
8000/8000 [==============================] - 108s 14ms/step - loss: 2.5310 - accuracy: 0.3655 - val_loss: 2.2824 - val_accuracy: 0.3505
Epoch 17/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.5138 - accuracy: 0.3688 - val_loss: 3.1626 - val_accuracy: 0.3591
Epoch 18/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.5127 - accuracy: 0.3709 - val_loss: 3.3342 - val_accuracy: 0.3162
Epoch 19/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.4977 - accuracy: 0.3756 - val_loss: 3.0506 - val_accuracy: 0.3142
Epoch 20/50
8000/8000 [==============================] - 109s 14ms/step - loss: 2.4897 - accuracy: 0.3765 - val_loss: 1.7744 - val_accuracy: 0.3700
Epoch 21/50
8000/8000 [==============================] - 110s 14ms/step - loss: 2.4821 - accuracy: 0.3797 - val_loss: 1.6011 - val_accuracy: 0.3888
Epoch 22/50
8000/8000 [==============================] - 110s 14ms/step - loss: 2.4794 - accuracy: 0.3805 - val_loss: 2.5584 - val_accuracy: 0.3813
Epoch 23/50
8000/8000 [==============================] - 110s 14ms/step - loss: 2.4738 - accuracy: 0.3815 - val_loss: 1.8191 - val_accuracy: 0.3747
Epoch 24/50
8000/8000 [==============================] - 108s 14ms/step - loss: 2.4666 - accuracy: 0.3840 - val_loss: 2.9815 - val_accuracy: 0.3706
Epoch 25/50
8000/8000 [==============================] - 108s 14ms/step - loss: 2.4527 - accuracy: 0.3884 - val_loss: 3.5520 - val_accuracy: 0.3771
Epoch 26/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.2213 - accuracy: 0.4302 - val_loss: 2.8158 - val_accuracy: 0.4091
Epoch 27/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.1724 - accuracy: 0.4428 - val_loss: 3.0203 - val_accuracy: 0.4137
Epoch 28/50
8000/8000 [==============================] - 109s 14ms/step - loss: 2.1413 - accuracy: 0.4503 - val_loss: 1.1151 - val_accuracy: 0.4157
Epoch 29/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.1195 - accuracy: 0.4525 - val_loss: 1.6667 - val_accuracy: 0.4101
Epoch 30/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.1008 - accuracy: 0.4576 - val_loss: 1.9740 - val_accuracy: 0.4206
Epoch 31/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.0926 - accuracy: 0.4588 - val_loss: 2.5314 - val_accuracy: 0.4096
Epoch 32/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.0790 - accuracy: 0.4622 - val_loss: 1.9146 - val_accuracy: 0.4147
Epoch 33/50
8000/8000 [==============================] - 108s 14ms/step - loss: 2.0567 - accuracy: 0.4643 - val_loss: 3.2467 - val_accuracy: 0.4209
Epoch 34/50
8000/8000 [==============================] - 109s 14ms/step - loss: 2.0481 - accuracy: 0.4681 - val_loss: 4.8671 - val_accuracy: 0.4195
Epoch 35/50
8000/8000 [==============================] - 108s 13ms/step - loss: 2.0507 - accuracy: 0.4676 - val_loss: 2.7162 - val_accuracy: 0.4198
Epoch 36/50
8000/8000 [==============================] - 109s 14ms/step - loss: 2.0410 - accuracy: 0.4695 - val_loss: 1.4512 - val_accuracy: 0.4230
Epoch 37/50
8000/8000 [==============================] - 109s 14ms/step - loss: 2.0457 - accuracy: 0.4686 - val_loss: 1.9649 - val_accuracy: 0.4235
Epoch 38/50
8000/8000 [==============================] - 110s 14ms/step - loss: 2.0407 - accuracy: 0.4701 - val_loss: 2.4972 - val_accuracy: 0.4214
Epoch 39/50
8000/8000 [==============================] - 110s 14ms/step - loss: 2.0426 - accuracy: 0.4706 - val_loss: 1.5233 - val_accuracy: 0.4225
Epoch 40/50
8000/8000 [==============================] - 110s 14ms/step - loss: 2.0373 - accuracy: 0.4690 - val_loss: 1.8184 - val_accuracy: 0.4184
Epoch 41/50
8000/8000 [==============================] - 110s 14ms/step - loss: 2.0393 - accuracy: 0.4704 - val_loss: 2.0175 - val_accuracy: 0.4171
Epoch 42/50
8000/8000 [==============================] - 111s 14ms/step - loss: 2.0405 - accuracy: 0.4705 - val_loss: 2.1127 - val_accuracy: 0.4223
Epoch 43/50
8000/8000 [==============================] - 110s 14ms/step - loss: 2.0495 - accuracy: 0.4683 - val_loss: 2.2789 - val_accuracy: 0.4191
In [0]:
# Plot training & validation accuracy rates
fig, ax = plt.subplots(1,2,figsize=(12,4))
ax[0].plot(history2.history['accuracy'])
ax[0].plot(history2.history['val_accuracy'])
ax[0].set_title('Model2 Accuracy')
ax[0].set_ylabel('Accuracy')
ax[0].set_xlabel('Epoch')
ax[0].legend(['Train', 'Test'], loc='upper left')

# Plot training & validation loss rates
ax[1].plot(history2.history['loss'])
ax[1].plot(history2.history['val_loss'])
ax[1].set_title('Model2 Loss *')
ax[1].set_ylabel('Loss')
ax[1].set_xlabel('Epoch')
ax[1].legend(['Train', 'Test'], loc='upper left')
plt.show()

This model's accuracy is simillar to the last model's, but the difference between the final train accuracy and validation accuracy is smaller than the corresponding scores on the last model, which means this model is less overfitting than the last.

*On the 'Model2 Loss' chart, the train loss is apparent to be 0, but it is because the initial peaks of the validation loss which are greater compared to the low values of the train loss

In [0]:
# plotting a heat map of the predictions on unseen data

y_pred = model2.predict(X_test)
pred_cat = np.argmax(y_pred,axis=1)
confusion_mat2 = confusion_matrix(y_test,pred_cat)
print('model accuracy on test set is: {}%'.format(accuracy_score(y_test,pred_cat)*100))
sns.heatmap(confusion_mat2,cmap='Greens',annot=True, fmt='d')
sns.set(rc={'figure.figsize':(50,30)})
plt.xlabel('Prediction')
plt.ylabel('True label')
plt.title('Second model results on test set')
model accuracy on test set is: 46.36%
Out[0]:
Text(0.5, 1.0, 'Second model results on test set')

Apparently, the improved model has problems with fish. The most common mistake was predicting Whales to be Dolphins.

Below are a few examples for samples from the corresponding classes.

In [21]:
# picking a common mistake and plotting corresponding samples

pred_index = 30 # dolphin
true_index = 95 # whale

#print samples of the relevant classes
plot_samples(pred_index, 3)
plot_samples(true_index, 3)

Part 3- Feature Extruction

We chose ResNet50 architecture with weights calculated for solving ImageNet

a. We excluded the last (top) layer and added a 100-neurons dense layer instead.

b. We are training all the weights of the model to fit it to our problem (rather then being fit for ImageNet)

In [4]:
from keras.applications.resnet import ResNet50

# instantiating a pre-weighted ResNet50 model without it's original top
rn50_model = ResNet50(weights='imagenet', include_top=False, pooling='max', input_shape=(32,32,3))

# adding a final layer for adjusting the model for the given 100 class problem
last_hidden_layer = rn50_model._layers[-1].output
new_output_layer = Dense(100, activation='softmax')(last_hidden_layer)
new_rn50_model = Model(rn50_model.inputs, [new_output_layer])

# preparing the model for training the last layer
new_rn50_model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])
new_rn50_model.summary()
Downloading data from https://github.com/keras-team/keras-applications/releases/download/resnet/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5
94773248/94765736 [==============================] - 8s 0us/step
Model: "model_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            (None, 32, 32, 3)    0                                            
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D)       (None, 38, 38, 3)    0           input_1[0][0]                    
__________________________________________________________________________________________________
conv1_conv (Conv2D)             (None, 16, 16, 64)   9472        conv1_pad[0][0]                  
__________________________________________________________________________________________________
conv1_bn (BatchNormalization)   (None, 16, 16, 64)   256         conv1_conv[0][0]                 
__________________________________________________________________________________________________
conv1_relu (Activation)         (None, 16, 16, 64)   0           conv1_bn[0][0]                   
__________________________________________________________________________________________________
pool1_pad (ZeroPadding2D)       (None, 18, 18, 64)   0           conv1_relu[0][0]                 
__________________________________________________________________________________________________
pool1_pool (MaxPooling2D)       (None, 8, 8, 64)     0           pool1_pad[0][0]                  
__________________________________________________________________________________________________
conv2_block1_1_conv (Conv2D)    (None, 8, 8, 64)     4160        pool1_pool[0][0]                 
__________________________________________________________________________________________________
conv2_block1_1_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv2_block1_1_relu (Activation (None, 8, 8, 64)     0           conv2_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv2_block1_2_conv (Conv2D)    (None, 8, 8, 64)     36928       conv2_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv2_block1_2_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv2_block1_2_relu (Activation (None, 8, 8, 64)     0           conv2_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv2_block1_0_conv (Conv2D)    (None, 8, 8, 256)    16640       pool1_pool[0][0]                 
__________________________________________________________________________________________________
conv2_block1_3_conv (Conv2D)    (None, 8, 8, 256)    16640       conv2_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv2_block1_0_bn (BatchNormali (None, 8, 8, 256)    1024        conv2_block1_0_conv[0][0]        
__________________________________________________________________________________________________
conv2_block1_3_bn (BatchNormali (None, 8, 8, 256)    1024        conv2_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv2_block1_add (Add)          (None, 8, 8, 256)    0           conv2_block1_0_bn[0][0]          
                                                                 conv2_block1_3_bn[0][0]          
__________________________________________________________________________________________________
conv2_block1_out (Activation)   (None, 8, 8, 256)    0           conv2_block1_add[0][0]           
__________________________________________________________________________________________________
conv2_block2_1_conv (Conv2D)    (None, 8, 8, 64)     16448       conv2_block1_out[0][0]           
__________________________________________________________________________________________________
conv2_block2_1_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv2_block2_1_relu (Activation (None, 8, 8, 64)     0           conv2_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv2_block2_2_conv (Conv2D)    (None, 8, 8, 64)     36928       conv2_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv2_block2_2_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv2_block2_2_relu (Activation (None, 8, 8, 64)     0           conv2_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv2_block2_3_conv (Conv2D)    (None, 8, 8, 256)    16640       conv2_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv2_block2_3_bn (BatchNormali (None, 8, 8, 256)    1024        conv2_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv2_block2_add (Add)          (None, 8, 8, 256)    0           conv2_block1_out[0][0]           
                                                                 conv2_block2_3_bn[0][0]          
__________________________________________________________________________________________________
conv2_block2_out (Activation)   (None, 8, 8, 256)    0           conv2_block2_add[0][0]           
__________________________________________________________________________________________________
conv2_block3_1_conv (Conv2D)    (None, 8, 8, 64)     16448       conv2_block2_out[0][0]           
__________________________________________________________________________________________________
conv2_block3_1_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv2_block3_1_relu (Activation (None, 8, 8, 64)     0           conv2_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv2_block3_2_conv (Conv2D)    (None, 8, 8, 64)     36928       conv2_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv2_block3_2_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv2_block3_2_relu (Activation (None, 8, 8, 64)     0           conv2_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv2_block3_3_conv (Conv2D)    (None, 8, 8, 256)    16640       conv2_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv2_block3_3_bn (BatchNormali (None, 8, 8, 256)    1024        conv2_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv2_block3_add (Add)          (None, 8, 8, 256)    0           conv2_block2_out[0][0]           
                                                                 conv2_block3_3_bn[0][0]          
__________________________________________________________________________________________________
conv2_block3_out (Activation)   (None, 8, 8, 256)    0           conv2_block3_add[0][0]           
__________________________________________________________________________________________________
conv3_block1_1_conv (Conv2D)    (None, 4, 4, 128)    32896       conv2_block3_out[0][0]           
__________________________________________________________________________________________________
conv3_block1_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_1_relu (Activation (None, 4, 4, 128)    0           conv3_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block1_2_conv (Conv2D)    (None, 4, 4, 128)    147584      conv3_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block1_2_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_2_relu (Activation (None, 4, 4, 128)    0           conv3_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block1_0_conv (Conv2D)    (None, 4, 4, 512)    131584      conv2_block3_out[0][0]           
__________________________________________________________________________________________________
conv3_block1_3_conv (Conv2D)    (None, 4, 4, 512)    66048       conv3_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block1_0_bn (BatchNormali (None, 4, 4, 512)    2048        conv3_block1_0_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_3_bn (BatchNormali (None, 4, 4, 512)    2048        conv3_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_add (Add)          (None, 4, 4, 512)    0           conv3_block1_0_bn[0][0]          
                                                                 conv3_block1_3_bn[0][0]          
__________________________________________________________________________________________________
conv3_block1_out (Activation)   (None, 4, 4, 512)    0           conv3_block1_add[0][0]           
__________________________________________________________________________________________________
conv3_block2_1_conv (Conv2D)    (None, 4, 4, 128)    65664       conv3_block1_out[0][0]           
__________________________________________________________________________________________________
conv3_block2_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block2_1_relu (Activation (None, 4, 4, 128)    0           conv3_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block2_2_conv (Conv2D)    (None, 4, 4, 128)    147584      conv3_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block2_2_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block2_2_relu (Activation (None, 4, 4, 128)    0           conv3_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block2_3_conv (Conv2D)    (None, 4, 4, 512)    66048       conv3_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block2_3_bn (BatchNormali (None, 4, 4, 512)    2048        conv3_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block2_add (Add)          (None, 4, 4, 512)    0           conv3_block1_out[0][0]           
                                                                 conv3_block2_3_bn[0][0]          
__________________________________________________________________________________________________
conv3_block2_out (Activation)   (None, 4, 4, 512)    0           conv3_block2_add[0][0]           
__________________________________________________________________________________________________
conv3_block3_1_conv (Conv2D)    (None, 4, 4, 128)    65664       conv3_block2_out[0][0]           
__________________________________________________________________________________________________
conv3_block3_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block3_1_relu (Activation (None, 4, 4, 128)    0           conv3_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block3_2_conv (Conv2D)    (None, 4, 4, 128)    147584      conv3_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block3_2_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block3_2_relu (Activation (None, 4, 4, 128)    0           conv3_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block3_3_conv (Conv2D)    (None, 4, 4, 512)    66048       conv3_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block3_3_bn (BatchNormali (None, 4, 4, 512)    2048        conv3_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block3_add (Add)          (None, 4, 4, 512)    0           conv3_block2_out[0][0]           
                                                                 conv3_block3_3_bn[0][0]          
__________________________________________________________________________________________________
conv3_block3_out (Activation)   (None, 4, 4, 512)    0           conv3_block3_add[0][0]           
__________________________________________________________________________________________________
conv3_block4_1_conv (Conv2D)    (None, 4, 4, 128)    65664       conv3_block3_out[0][0]           
__________________________________________________________________________________________________
conv3_block4_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block4_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block4_1_relu (Activation (None, 4, 4, 128)    0           conv3_block4_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block4_2_conv (Conv2D)    (None, 4, 4, 128)    147584      conv3_block4_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block4_2_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block4_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block4_2_relu (Activation (None, 4, 4, 128)    0           conv3_block4_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block4_3_conv (Conv2D)    (None, 4, 4, 512)    66048       conv3_block4_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block4_3_bn (BatchNormali (None, 4, 4, 512)    2048        conv3_block4_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block4_add (Add)          (None, 4, 4, 512)    0           conv3_block3_out[0][0]           
                                                                 conv3_block4_3_bn[0][0]          
__________________________________________________________________________________________________
conv3_block4_out (Activation)   (None, 4, 4, 512)    0           conv3_block4_add[0][0]           
__________________________________________________________________________________________________
conv4_block1_1_conv (Conv2D)    (None, 2, 2, 256)    131328      conv3_block4_out[0][0]           
__________________________________________________________________________________________________
conv4_block1_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_1_relu (Activation (None, 2, 2, 256)    0           conv4_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block1_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block1_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_2_relu (Activation (None, 2, 2, 256)    0           conv4_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block1_0_conv (Conv2D)    (None, 2, 2, 1024)   525312      conv3_block4_out[0][0]           
__________________________________________________________________________________________________
conv4_block1_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block1_0_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block1_0_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_add (Add)          (None, 2, 2, 1024)   0           conv4_block1_0_bn[0][0]          
                                                                 conv4_block1_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block1_out (Activation)   (None, 2, 2, 1024)   0           conv4_block1_add[0][0]           
__________________________________________________________________________________________________
conv4_block2_1_conv (Conv2D)    (None, 2, 2, 256)    262400      conv4_block1_out[0][0]           
__________________________________________________________________________________________________
conv4_block2_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block2_1_relu (Activation (None, 2, 2, 256)    0           conv4_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block2_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block2_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block2_2_relu (Activation (None, 2, 2, 256)    0           conv4_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block2_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block2_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block2_add (Add)          (None, 2, 2, 1024)   0           conv4_block1_out[0][0]           
                                                                 conv4_block2_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block2_out (Activation)   (None, 2, 2, 1024)   0           conv4_block2_add[0][0]           
__________________________________________________________________________________________________
conv4_block3_1_conv (Conv2D)    (None, 2, 2, 256)    262400      conv4_block2_out[0][0]           
__________________________________________________________________________________________________
conv4_block3_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block3_1_relu (Activation (None, 2, 2, 256)    0           conv4_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block3_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block3_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block3_2_relu (Activation (None, 2, 2, 256)    0           conv4_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block3_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block3_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block3_add (Add)          (None, 2, 2, 1024)   0           conv4_block2_out[0][0]           
                                                                 conv4_block3_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block3_out (Activation)   (None, 2, 2, 1024)   0           conv4_block3_add[0][0]           
__________________________________________________________________________________________________
conv4_block4_1_conv (Conv2D)    (None, 2, 2, 256)    262400      conv4_block3_out[0][0]           
__________________________________________________________________________________________________
conv4_block4_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block4_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block4_1_relu (Activation (None, 2, 2, 256)    0           conv4_block4_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block4_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block4_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block4_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block4_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block4_2_relu (Activation (None, 2, 2, 256)    0           conv4_block4_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block4_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block4_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block4_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block4_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block4_add (Add)          (None, 2, 2, 1024)   0           conv4_block3_out[0][0]           
                                                                 conv4_block4_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block4_out (Activation)   (None, 2, 2, 1024)   0           conv4_block4_add[0][0]           
__________________________________________________________________________________________________
conv4_block5_1_conv (Conv2D)    (None, 2, 2, 256)    262400      conv4_block4_out[0][0]           
__________________________________________________________________________________________________
conv4_block5_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block5_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block5_1_relu (Activation (None, 2, 2, 256)    0           conv4_block5_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block5_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block5_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block5_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block5_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block5_2_relu (Activation (None, 2, 2, 256)    0           conv4_block5_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block5_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block5_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block5_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block5_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block5_add (Add)          (None, 2, 2, 1024)   0           conv4_block4_out[0][0]           
                                                                 conv4_block5_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block5_out (Activation)   (None, 2, 2, 1024)   0           conv4_block5_add[0][0]           
__________________________________________________________________________________________________
conv4_block6_1_conv (Conv2D)    (None, 2, 2, 256)    262400      conv4_block5_out[0][0]           
__________________________________________________________________________________________________
conv4_block6_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block6_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block6_1_relu (Activation (None, 2, 2, 256)    0           conv4_block6_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block6_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block6_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block6_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block6_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block6_2_relu (Activation (None, 2, 2, 256)    0           conv4_block6_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block6_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block6_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block6_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block6_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block6_add (Add)          (None, 2, 2, 1024)   0           conv4_block5_out[0][0]           
                                                                 conv4_block6_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block6_out (Activation)   (None, 2, 2, 1024)   0           conv4_block6_add[0][0]           
__________________________________________________________________________________________________
conv5_block1_1_conv (Conv2D)    (None, 1, 1, 512)    524800      conv4_block6_out[0][0]           
__________________________________________________________________________________________________
conv5_block1_1_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv5_block1_1_relu (Activation (None, 1, 1, 512)    0           conv5_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv5_block1_2_conv (Conv2D)    (None, 1, 1, 512)    2359808     conv5_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv5_block1_2_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv5_block1_2_relu (Activation (None, 1, 1, 512)    0           conv5_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv5_block1_0_conv (Conv2D)    (None, 1, 1, 2048)   2099200     conv4_block6_out[0][0]           
__________________________________________________________________________________________________
conv5_block1_3_conv (Conv2D)    (None, 1, 1, 2048)   1050624     conv5_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block1_0_bn (BatchNormali (None, 1, 1, 2048)   8192        conv5_block1_0_conv[0][0]        
__________________________________________________________________________________________________
conv5_block1_3_bn (BatchNormali (None, 1, 1, 2048)   8192        conv5_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv5_block1_add (Add)          (None, 1, 1, 2048)   0           conv5_block1_0_bn[0][0]          
                                                                 conv5_block1_3_bn[0][0]          
__________________________________________________________________________________________________
conv5_block1_out (Activation)   (None, 1, 1, 2048)   0           conv5_block1_add[0][0]           
__________________________________________________________________________________________________
conv5_block2_1_conv (Conv2D)    (None, 1, 1, 512)    1049088     conv5_block1_out[0][0]           
__________________________________________________________________________________________________
conv5_block2_1_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv5_block2_1_relu (Activation (None, 1, 1, 512)    0           conv5_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv5_block2_2_conv (Conv2D)    (None, 1, 1, 512)    2359808     conv5_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv5_block2_2_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv5_block2_2_relu (Activation (None, 1, 1, 512)    0           conv5_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv5_block2_3_conv (Conv2D)    (None, 1, 1, 2048)   1050624     conv5_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block2_3_bn (BatchNormali (None, 1, 1, 2048)   8192        conv5_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv5_block2_add (Add)          (None, 1, 1, 2048)   0           conv5_block1_out[0][0]           
                                                                 conv5_block2_3_bn[0][0]          
__________________________________________________________________________________________________
conv5_block2_out (Activation)   (None, 1, 1, 2048)   0           conv5_block2_add[0][0]           
__________________________________________________________________________________________________
conv5_block3_1_conv (Conv2D)    (None, 1, 1, 512)    1049088     conv5_block2_out[0][0]           
__________________________________________________________________________________________________
conv5_block3_1_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_1_relu (Activation (None, 1, 1, 512)    0           conv5_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv5_block3_2_conv (Conv2D)    (None, 1, 1, 512)    2359808     conv5_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv5_block3_2_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_2_relu (Activation (None, 1, 1, 512)    0           conv5_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv5_block3_3_conv (Conv2D)    (None, 1, 1, 2048)   1050624     conv5_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block3_3_bn (BatchNormali (None, 1, 1, 2048)   8192        conv5_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_add (Add)          (None, 1, 1, 2048)   0           conv5_block2_out[0][0]           
                                                                 conv5_block3_3_bn[0][0]          
__________________________________________________________________________________________________
conv5_block3_out (Activation)   (None, 1, 1, 2048)   0           conv5_block3_add[0][0]           
__________________________________________________________________________________________________
max_pool (GlobalMaxPooling2D)   (None, 2048)         0           conv5_block3_out[0][0]           
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 100)          204900      max_pool[0][0]                   
==================================================================================================
Total params: 23,792,612
Trainable params: 23,739,492
Non-trainable params: 53,120
__________________________________________________________________________________________________

c. Fitting the model with the new layer on top

In [5]:
# fitting the model
monitor_val_acc = EarlyStopping(monitor='val_accuracy', patience=6)
lr_sched = ReduceLROnPlateau(patience=4)

new_rn50_model.fit(X_train,
    to_categorical(y_train),
    batch_size=200,
    epochs=20,
    verbose=1,
    shuffle=True,
    callbacks=[monitor_val_acc, lr_sched],
    validation_split=0.2)
Train on 40000 samples, validate on 10000 samples
Epoch 1/20
40000/40000 [==============================] - 103s 3ms/step - loss: 4.0686 - accuracy: 0.1529 - val_loss: 9.7896 - val_accuracy: 0.0933
Epoch 2/20
40000/40000 [==============================] - 78s 2ms/step - loss: 3.1891 - accuracy: 0.2778 - val_loss: 7.4427 - val_accuracy: 0.2017
Epoch 3/20
40000/40000 [==============================] - 78s 2ms/step - loss: 2.7958 - accuracy: 0.3460 - val_loss: 3.5436 - val_accuracy: 0.2529
Epoch 4/20
40000/40000 [==============================] - 78s 2ms/step - loss: 2.7475 - accuracy: 0.3911 - val_loss: 4.2702 - val_accuracy: 0.2134
Epoch 5/20
40000/40000 [==============================] - 78s 2ms/step - loss: 2.3361 - accuracy: 0.4044 - val_loss: 2.7826 - val_accuracy: 0.3242
Epoch 6/20
40000/40000 [==============================] - 78s 2ms/step - loss: 1.9043 - accuracy: 0.4996 - val_loss: 2.8031 - val_accuracy: 0.3392
Epoch 7/20
40000/40000 [==============================] - 78s 2ms/step - loss: 1.5758 - accuracy: 0.5645 - val_loss: 2.4210 - val_accuracy: 0.4105
Epoch 8/20
40000/40000 [==============================] - 78s 2ms/step - loss: 1.2514 - accuracy: 0.6401 - val_loss: 2.5031 - val_accuracy: 0.4163
Epoch 9/20
40000/40000 [==============================] - 78s 2ms/step - loss: 1.0152 - accuracy: 0.7025 - val_loss: 2.3874 - val_accuracy: 0.4393
Epoch 10/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.8242 - accuracy: 0.7588 - val_loss: 2.5651 - val_accuracy: 0.4391
Epoch 11/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.5721 - accuracy: 0.8307 - val_loss: 3.0740 - val_accuracy: 0.3847
Epoch 12/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.4881 - accuracy: 0.8543 - val_loss: 2.9906 - val_accuracy: 0.4259
Epoch 13/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.2756 - accuracy: 0.9195 - val_loss: 2.8915 - val_accuracy: 0.4389
Epoch 14/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.1067 - accuracy: 0.9765 - val_loss: 2.6263 - val_accuracy: 0.4866
Epoch 15/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.0543 - accuracy: 0.9928 - val_loss: 2.5906 - val_accuracy: 0.4987
Epoch 16/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.0411 - accuracy: 0.9958 - val_loss: 2.6207 - val_accuracy: 0.5015
Epoch 17/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.0314 - accuracy: 0.9969 - val_loss: 2.6394 - val_accuracy: 0.5019
Epoch 18/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.0277 - accuracy: 0.9981 - val_loss: 2.6535 - val_accuracy: 0.5028
Epoch 19/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.0255 - accuracy: 0.9981 - val_loss: 2.6581 - val_accuracy: 0.5025
Epoch 20/20
40000/40000 [==============================] - 78s 2ms/step - loss: 0.0238 - accuracy: 0.9984 - val_loss: 2.6590 - val_accuracy: 0.5036
Out[5]:
<keras.callbacks.callbacks.History at 0x7fa7c7d1ca90>
In [22]:
# plotting a heat map of the predictions on unseen data

y_pred = new_rn50_model.predict(X_test)
pred_cat = np.argmax(y_pred,axis=1)
confusion_mat2 = confusion_matrix(y_test,pred_cat)
print('model accuracy on test set is: {}%'.format(accuracy_score(y_test,pred_cat)*100))
sns.heatmap(confusion_mat2,cmap='Greens',annot=True, fmt='d')
sns.set(rc={'figure.figsize':(50,30)})
plt.xlabel('Prediction')
plt.ylabel('True label')
plt.title('ResNet50 model results on test set')
model accuracy on test set is: 50.6%
Out[22]:
Text(0.5, 1.0, 'ResNet50 model results on test set')

Finally, use the model as a feature extructor.

For this, the new layer should be removed, hence the features are "exposed"

In [23]:
# removing the last layer for exposing the features
new_rn50_model._layers.pop()
last_layer = new_rn50_model._layers[-1]
feature_extraction_model = Model(new_rn50_model.inputs, [last_layer.output])
feature_extraction_model.compile(loss='categorical_crossentropy',optimizer='adam')
feature_extraction_model.summary()

# producing predictions as features
train_features = feature_extraction_model.predict(X_train)
test_features = feature_extraction_model.predict(X_test)
Model: "model_2"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            (None, 32, 32, 3)    0                                            
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D)       (None, 38, 38, 3)    0           input_1[0][0]                    
__________________________________________________________________________________________________
conv1_conv (Conv2D)             (None, 16, 16, 64)   9472        conv1_pad[0][0]                  
__________________________________________________________________________________________________
conv1_bn (BatchNormalization)   (None, 16, 16, 64)   256         conv1_conv[0][0]                 
__________________________________________________________________________________________________
conv1_relu (Activation)         (None, 16, 16, 64)   0           conv1_bn[0][0]                   
__________________________________________________________________________________________________
pool1_pad (ZeroPadding2D)       (None, 18, 18, 64)   0           conv1_relu[0][0]                 
__________________________________________________________________________________________________
pool1_pool (MaxPooling2D)       (None, 8, 8, 64)     0           pool1_pad[0][0]                  
__________________________________________________________________________________________________
conv2_block1_1_conv (Conv2D)    (None, 8, 8, 64)     4160        pool1_pool[0][0]                 
__________________________________________________________________________________________________
conv2_block1_1_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv2_block1_1_relu (Activation (None, 8, 8, 64)     0           conv2_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv2_block1_2_conv (Conv2D)    (None, 8, 8, 64)     36928       conv2_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv2_block1_2_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv2_block1_2_relu (Activation (None, 8, 8, 64)     0           conv2_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv2_block1_0_conv (Conv2D)    (None, 8, 8, 256)    16640       pool1_pool[0][0]                 
__________________________________________________________________________________________________
conv2_block1_3_conv (Conv2D)    (None, 8, 8, 256)    16640       conv2_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv2_block1_0_bn (BatchNormali (None, 8, 8, 256)    1024        conv2_block1_0_conv[0][0]        
__________________________________________________________________________________________________
conv2_block1_3_bn (BatchNormali (None, 8, 8, 256)    1024        conv2_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv2_block1_add (Add)          (None, 8, 8, 256)    0           conv2_block1_0_bn[0][0]          
                                                                 conv2_block1_3_bn[0][0]          
__________________________________________________________________________________________________
conv2_block1_out (Activation)   (None, 8, 8, 256)    0           conv2_block1_add[0][0]           
__________________________________________________________________________________________________
conv2_block2_1_conv (Conv2D)    (None, 8, 8, 64)     16448       conv2_block1_out[0][0]           
__________________________________________________________________________________________________
conv2_block2_1_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv2_block2_1_relu (Activation (None, 8, 8, 64)     0           conv2_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv2_block2_2_conv (Conv2D)    (None, 8, 8, 64)     36928       conv2_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv2_block2_2_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv2_block2_2_relu (Activation (None, 8, 8, 64)     0           conv2_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv2_block2_3_conv (Conv2D)    (None, 8, 8, 256)    16640       conv2_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv2_block2_3_bn (BatchNormali (None, 8, 8, 256)    1024        conv2_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv2_block2_add (Add)          (None, 8, 8, 256)    0           conv2_block1_out[0][0]           
                                                                 conv2_block2_3_bn[0][0]          
__________________________________________________________________________________________________
conv2_block2_out (Activation)   (None, 8, 8, 256)    0           conv2_block2_add[0][0]           
__________________________________________________________________________________________________
conv2_block3_1_conv (Conv2D)    (None, 8, 8, 64)     16448       conv2_block2_out[0][0]           
__________________________________________________________________________________________________
conv2_block3_1_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv2_block3_1_relu (Activation (None, 8, 8, 64)     0           conv2_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv2_block3_2_conv (Conv2D)    (None, 8, 8, 64)     36928       conv2_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv2_block3_2_bn (BatchNormali (None, 8, 8, 64)     256         conv2_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv2_block3_2_relu (Activation (None, 8, 8, 64)     0           conv2_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv2_block3_3_conv (Conv2D)    (None, 8, 8, 256)    16640       conv2_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv2_block3_3_bn (BatchNormali (None, 8, 8, 256)    1024        conv2_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv2_block3_add (Add)          (None, 8, 8, 256)    0           conv2_block2_out[0][0]           
                                                                 conv2_block3_3_bn[0][0]          
__________________________________________________________________________________________________
conv2_block3_out (Activation)   (None, 8, 8, 256)    0           conv2_block3_add[0][0]           
__________________________________________________________________________________________________
conv3_block1_1_conv (Conv2D)    (None, 4, 4, 128)    32896       conv2_block3_out[0][0]           
__________________________________________________________________________________________________
conv3_block1_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_1_relu (Activation (None, 4, 4, 128)    0           conv3_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block1_2_conv (Conv2D)    (None, 4, 4, 128)    147584      conv3_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block1_2_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_2_relu (Activation (None, 4, 4, 128)    0           conv3_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block1_0_conv (Conv2D)    (None, 4, 4, 512)    131584      conv2_block3_out[0][0]           
__________________________________________________________________________________________________
conv3_block1_3_conv (Conv2D)    (None, 4, 4, 512)    66048       conv3_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block1_0_bn (BatchNormali (None, 4, 4, 512)    2048        conv3_block1_0_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_3_bn (BatchNormali (None, 4, 4, 512)    2048        conv3_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_add (Add)          (None, 4, 4, 512)    0           conv3_block1_0_bn[0][0]          
                                                                 conv3_block1_3_bn[0][0]          
__________________________________________________________________________________________________
conv3_block1_out (Activation)   (None, 4, 4, 512)    0           conv3_block1_add[0][0]           
__________________________________________________________________________________________________
conv3_block2_1_conv (Conv2D)    (None, 4, 4, 128)    65664       conv3_block1_out[0][0]           
__________________________________________________________________________________________________
conv3_block2_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block2_1_relu (Activation (None, 4, 4, 128)    0           conv3_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block2_2_conv (Conv2D)    (None, 4, 4, 128)    147584      conv3_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block2_2_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block2_2_relu (Activation (None, 4, 4, 128)    0           conv3_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block2_3_conv (Conv2D)    (None, 4, 4, 512)    66048       conv3_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block2_3_bn (BatchNormali (None, 4, 4, 512)    2048        conv3_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block2_add (Add)          (None, 4, 4, 512)    0           conv3_block1_out[0][0]           
                                                                 conv3_block2_3_bn[0][0]          
__________________________________________________________________________________________________
conv3_block2_out (Activation)   (None, 4, 4, 512)    0           conv3_block2_add[0][0]           
__________________________________________________________________________________________________
conv3_block3_1_conv (Conv2D)    (None, 4, 4, 128)    65664       conv3_block2_out[0][0]           
__________________________________________________________________________________________________
conv3_block3_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block3_1_relu (Activation (None, 4, 4, 128)    0           conv3_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block3_2_conv (Conv2D)    (None, 4, 4, 128)    147584      conv3_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block3_2_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block3_2_relu (Activation (None, 4, 4, 128)    0           conv3_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block3_3_conv (Conv2D)    (None, 4, 4, 512)    66048       conv3_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block3_3_bn (BatchNormali (None, 4, 4, 512)    2048        conv3_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block3_add (Add)          (None, 4, 4, 512)    0           conv3_block2_out[0][0]           
                                                                 conv3_block3_3_bn[0][0]          
__________________________________________________________________________________________________
conv3_block3_out (Activation)   (None, 4, 4, 512)    0           conv3_block3_add[0][0]           
__________________________________________________________________________________________________
conv3_block4_1_conv (Conv2D)    (None, 4, 4, 128)    65664       conv3_block3_out[0][0]           
__________________________________________________________________________________________________
conv3_block4_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block4_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block4_1_relu (Activation (None, 4, 4, 128)    0           conv3_block4_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block4_2_conv (Conv2D)    (None, 4, 4, 128)    147584      conv3_block4_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block4_2_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block4_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block4_2_relu (Activation (None, 4, 4, 128)    0           conv3_block4_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block4_3_conv (Conv2D)    (None, 4, 4, 512)    66048       conv3_block4_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block4_3_bn (BatchNormali (None, 4, 4, 512)    2048        conv3_block4_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block4_add (Add)          (None, 4, 4, 512)    0           conv3_block3_out[0][0]           
                                                                 conv3_block4_3_bn[0][0]          
__________________________________________________________________________________________________
conv3_block4_out (Activation)   (None, 4, 4, 512)    0           conv3_block4_add[0][0]           
__________________________________________________________________________________________________
conv4_block1_1_conv (Conv2D)    (None, 2, 2, 256)    131328      conv3_block4_out[0][0]           
__________________________________________________________________________________________________
conv4_block1_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_1_relu (Activation (None, 2, 2, 256)    0           conv4_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block1_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block1_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_2_relu (Activation (None, 2, 2, 256)    0           conv4_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block1_0_conv (Conv2D)    (None, 2, 2, 1024)   525312      conv3_block4_out[0][0]           
__________________________________________________________________________________________________
conv4_block1_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block1_0_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block1_0_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_add (Add)          (None, 2, 2, 1024)   0           conv4_block1_0_bn[0][0]          
                                                                 conv4_block1_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block1_out (Activation)   (None, 2, 2, 1024)   0           conv4_block1_add[0][0]           
__________________________________________________________________________________________________
conv4_block2_1_conv (Conv2D)    (None, 2, 2, 256)    262400      conv4_block1_out[0][0]           
__________________________________________________________________________________________________
conv4_block2_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block2_1_relu (Activation (None, 2, 2, 256)    0           conv4_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block2_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block2_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block2_2_relu (Activation (None, 2, 2, 256)    0           conv4_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block2_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block2_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block2_add (Add)          (None, 2, 2, 1024)   0           conv4_block1_out[0][0]           
                                                                 conv4_block2_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block2_out (Activation)   (None, 2, 2, 1024)   0           conv4_block2_add[0][0]           
__________________________________________________________________________________________________
conv4_block3_1_conv (Conv2D)    (None, 2, 2, 256)    262400      conv4_block2_out[0][0]           
__________________________________________________________________________________________________
conv4_block3_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block3_1_relu (Activation (None, 2, 2, 256)    0           conv4_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block3_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block3_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block3_2_relu (Activation (None, 2, 2, 256)    0           conv4_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block3_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block3_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block3_add (Add)          (None, 2, 2, 1024)   0           conv4_block2_out[0][0]           
                                                                 conv4_block3_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block3_out (Activation)   (None, 2, 2, 1024)   0           conv4_block3_add[0][0]           
__________________________________________________________________________________________________
conv4_block4_1_conv (Conv2D)    (None, 2, 2, 256)    262400      conv4_block3_out[0][0]           
__________________________________________________________________________________________________
conv4_block4_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block4_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block4_1_relu (Activation (None, 2, 2, 256)    0           conv4_block4_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block4_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block4_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block4_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block4_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block4_2_relu (Activation (None, 2, 2, 256)    0           conv4_block4_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block4_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block4_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block4_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block4_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block4_add (Add)          (None, 2, 2, 1024)   0           conv4_block3_out[0][0]           
                                                                 conv4_block4_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block4_out (Activation)   (None, 2, 2, 1024)   0           conv4_block4_add[0][0]           
__________________________________________________________________________________________________
conv4_block5_1_conv (Conv2D)    (None, 2, 2, 256)    262400      conv4_block4_out[0][0]           
__________________________________________________________________________________________________
conv4_block5_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block5_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block5_1_relu (Activation (None, 2, 2, 256)    0           conv4_block5_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block5_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block5_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block5_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block5_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block5_2_relu (Activation (None, 2, 2, 256)    0           conv4_block5_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block5_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block5_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block5_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block5_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block5_add (Add)          (None, 2, 2, 1024)   0           conv4_block4_out[0][0]           
                                                                 conv4_block5_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block5_out (Activation)   (None, 2, 2, 1024)   0           conv4_block5_add[0][0]           
__________________________________________________________________________________________________
conv4_block6_1_conv (Conv2D)    (None, 2, 2, 256)    262400      conv4_block5_out[0][0]           
__________________________________________________________________________________________________
conv4_block6_1_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block6_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block6_1_relu (Activation (None, 2, 2, 256)    0           conv4_block6_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block6_2_conv (Conv2D)    (None, 2, 2, 256)    590080      conv4_block6_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block6_2_bn (BatchNormali (None, 2, 2, 256)    1024        conv4_block6_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block6_2_relu (Activation (None, 2, 2, 256)    0           conv4_block6_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block6_3_conv (Conv2D)    (None, 2, 2, 1024)   263168      conv4_block6_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block6_3_bn (BatchNormali (None, 2, 2, 1024)   4096        conv4_block6_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block6_add (Add)          (None, 2, 2, 1024)   0           conv4_block5_out[0][0]           
                                                                 conv4_block6_3_bn[0][0]          
__________________________________________________________________________________________________
conv4_block6_out (Activation)   (None, 2, 2, 1024)   0           conv4_block6_add[0][0]           
__________________________________________________________________________________________________
conv5_block1_1_conv (Conv2D)    (None, 1, 1, 512)    524800      conv4_block6_out[0][0]           
__________________________________________________________________________________________________
conv5_block1_1_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv5_block1_1_relu (Activation (None, 1, 1, 512)    0           conv5_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv5_block1_2_conv (Conv2D)    (None, 1, 1, 512)    2359808     conv5_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv5_block1_2_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv5_block1_2_relu (Activation (None, 1, 1, 512)    0           conv5_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv5_block1_0_conv (Conv2D)    (None, 1, 1, 2048)   2099200     conv4_block6_out[0][0]           
__________________________________________________________________________________________________
conv5_block1_3_conv (Conv2D)    (None, 1, 1, 2048)   1050624     conv5_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block1_0_bn (BatchNormali (None, 1, 1, 2048)   8192        conv5_block1_0_conv[0][0]        
__________________________________________________________________________________________________
conv5_block1_3_bn (BatchNormali (None, 1, 1, 2048)   8192        conv5_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv5_block1_add (Add)          (None, 1, 1, 2048)   0           conv5_block1_0_bn[0][0]          
                                                                 conv5_block1_3_bn[0][0]          
__________________________________________________________________________________________________
conv5_block1_out (Activation)   (None, 1, 1, 2048)   0           conv5_block1_add[0][0]           
__________________________________________________________________________________________________
conv5_block2_1_conv (Conv2D)    (None, 1, 1, 512)    1049088     conv5_block1_out[0][0]           
__________________________________________________________________________________________________
conv5_block2_1_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv5_block2_1_relu (Activation (None, 1, 1, 512)    0           conv5_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv5_block2_2_conv (Conv2D)    (None, 1, 1, 512)    2359808     conv5_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv5_block2_2_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv5_block2_2_relu (Activation (None, 1, 1, 512)    0           conv5_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv5_block2_3_conv (Conv2D)    (None, 1, 1, 2048)   1050624     conv5_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block2_3_bn (BatchNormali (None, 1, 1, 2048)   8192        conv5_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv5_block2_add (Add)          (None, 1, 1, 2048)   0           conv5_block1_out[0][0]           
                                                                 conv5_block2_3_bn[0][0]          
__________________________________________________________________________________________________
conv5_block2_out (Activation)   (None, 1, 1, 2048)   0           conv5_block2_add[0][0]           
__________________________________________________________________________________________________
conv5_block3_1_conv (Conv2D)    (None, 1, 1, 512)    1049088     conv5_block2_out[0][0]           
__________________________________________________________________________________________________
conv5_block3_1_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_1_relu (Activation (None, 1, 1, 512)    0           conv5_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv5_block3_2_conv (Conv2D)    (None, 1, 1, 512)    2359808     conv5_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv5_block3_2_bn (BatchNormali (None, 1, 1, 512)    2048        conv5_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_2_relu (Activation (None, 1, 1, 512)    0           conv5_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv5_block3_3_conv (Conv2D)    (None, 1, 1, 2048)   1050624     conv5_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block3_3_bn (BatchNormali (None, 1, 1, 2048)   8192        conv5_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_add (Add)          (None, 1, 1, 2048)   0           conv5_block2_out[0][0]           
                                                                 conv5_block3_3_bn[0][0]          
__________________________________________________________________________________________________
conv5_block3_out (Activation)   (None, 1, 1, 2048)   0           conv5_block3_add[0][0]           
__________________________________________________________________________________________________
max_pool (GlobalMaxPooling2D)   (None, 2048)         0           conv5_block3_out[0][0]           
==================================================================================================
Total params: 23,587,712
Trainable params: 23,534,592
Non-trainable params: 53,120
__________________________________________________________________________________________________

We chose Gaussian Naive Bayes for our classic ML model for proccessing the features.

In [24]:
from sklearn.naive_bayes import GaussianNB

gnb = GaussianNB()
#fitting the classic ML model on the features
gnb.fit(train_features, y_train.ravel())
Out[24]:
GaussianNB(priors=None, var_smoothing=1e-09)
In [25]:
# plotting a heat map of the predictions on unseen data

y_pred = gnb.predict(test_features)
confusion_mat = confusion_matrix(y_test,y_pred)
print('model accuracy on test set is: {}%'.format(accuracy_score(y_test,y_pred)*100))
sns.heatmap(confusion_mat,cmap='Greens',annot=True, fmt='d')
sns.set(rc={'figure.figsize':(50,30)})
plt.xlabel('Prediction')
plt.ylabel('True label')
plt.title('Feature extruction model results on test set')
model accuracy on test set is: 39.879999999999995%
Out[25]:
Text(0.5, 1.0, 'Feature extruction model results on test set')

We can see that the feature extraction did not surpass the ResNet50 model in validation accuracy, getting only 40% while the ResNet50 model got almost 51%.

Part 4- Report

These has been our first steps in the field of deep learning, which we've learned alot from.

Along the way we mostly experimented the basic concepts of developing a NN model, including trying our own ideas of model architectures, trying different widths of layers, trying different types of layers and different depth of architectures. after we got a sense for how it affects the duration and quality of the fitting, we aproached messing with the hyper-parameters in order to see the effect of those on the training process. A good example for this is trying to get above 1% of validation accuracy for hours, and then replacing the RAdam optimazer in Adam, suddenly getting much better results. This got us deep into understanding the diffrences between these two and the way each works.

In the process of estimating each model's performance we visualized and analysed the fitting process and the results of the process. While doing so, we learned the importance of this step. For example, from the accuracy/epoch chart it is easy to infer that the model is overfitted, and from the heat-map of the confusion matrix you can spot the most common mistakes the model made during classification of the test set. In our case, we saw that the first model had hard time differentiating between two kinds of trees and the second model mistakenly classified whales as dolphins, and these mistakes made us wonder what made the model make these mistakes. In the case of the trees, it is probably the general hue of the images and the texture of the foliage of the trees. When we examine the pictures of the whales and the dolphins side to side, the hues of the pictures are not so similar , which leads to the conclusion that the common feature is probably edge curves, that is, the body shape of a whale is similar to a dolphin's shape, disregarding the scale.

The use of good practice tools in training a good model also made us learn about the process. We saw that data augmentation is efficient in preventing overfitting and that feature extraction can be used to apply solutions that proven to work on other problems on our problem, which could be useful. In our case, the feature extraction didn't go as expected and the performence were not improved in comparison to the trained ResNet50 model. Prior to the feature extraction documented on this report, we tried fine tunning the ResNet50 model with it's weights fit to ImageNet, but it didn't went well eighter, achieving poor performance on the test set. In the procces of achieving better results, we tried also VGG16 as a feature extructor, and as classic ML model we tried KNeighbors, DecisionTree and SVC before setteling on GaussianNaiveBayes that got the best results.

Overall, we gained much knowledge (atleast in relation to what we held before) and now we feel more prepared for the challenges of the course that are yet to come.